{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f48a755a",
   "metadata": {
    "origin_pos": 0
   },
   "source": [
    "# 卷积神经网络（LeNet）\n",
    "\n",
    "通过之前几节，我们学习了构建一个完整卷积神经网络的所需组件：卷积、填充与步幅、池化。\n",
    "\n",
    "本节将介绍**LeNet**，它是最早发布的卷积神经网络之一。\n",
    "这个模型是由AT&T贝尔实验室的研究员**Yann LeCun**在1989年提出的（并以其命名），目的是识别图像中的手写数字。\n",
    "当时，Yann LeCun发表了第一篇通过反向传播成功训练卷积神经网络的研究，这项工作代表了十多年来神经网络研究开发的成果。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b931456a",
   "metadata": {},
   "source": [
    "## LeNet\n",
    "\n",
    "总体来看，(**LeNet（LeNet-5）由两个部分组成：**)\n",
    "\n",
    "* 卷积编码器：由两个卷积层组成;\n",
    "* 全连接层密集块：由三个全连接层组成。\n",
    "\n",
    "该架构如图所示。\n",
    "\n",
    "![LeNet中的数据流。输入是手写数字，输出为10种可能结果的概率。](../img/lenet.svg)\n",
    "\n",
    "每个卷积块中的基本单元是一个卷积层、一个sigmoid激活函数和平均汇聚层。请注意，虽然ReLU和最大汇聚层更有效，但它们在20世纪90年代还没有出现。每个卷积层使用$5\\times 5$卷积核和一个sigmoid激活函数。这些层将输入映射到多个二维特征输出，通常同时增加通道的数量。第一卷积层有6个输出通道，而第二个卷积层有16个输出通道。每个$2\\times2$池操作（步幅2）通过空间下采样将维数减少4倍。卷积的输出形状由批量大小、通道数、高度、宽度决定。\n",
    "\n",
    "为了将卷积块的输出传递给全连接稠密块，我们必须在小批量中展平每个样本。换言之，我们将这个四维输入转换成全连接层所期望的二维输入。这里的二维表示的第一个维度索引小批量中的样本，第二个维度给出每个样本的平面向量表示。LeNet的稠密块有三个全连接层，分别有120、84和10个输出。因为我们在执行分类任务，所以输出层的10维对应于最后输出结果的数量。\n",
    "\n",
    "通过下面的LeNet代码，可以看出用深度学习框架实现此类模型非常简单。我们只需要实例化一个`Sequential`块并将需要的层连接在一起。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "f6f29f8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 导入PyTorch\n",
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "bf8e9751",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T17:38:35.894137Z",
     "iopub.status.busy": "2022-12-07T17:38:35.893495Z",
     "iopub.status.idle": "2022-12-07T17:38:38.403653Z",
     "shell.execute_reply": "2022-12-07T17:38:38.402822Z"
    },
    "origin_pos": 2,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "# 用Sequential组织LeNet-5的网络模型net\n",
    "net = nn.Sequential(\n",
    "    # 卷积运算：1个输入通道，6个输出通道，5*5卷积核，填充2，\n",
    "    nn.Conv2d(1,6,kernel_size=5,padding=2),\n",
    "    # 激活函数为Sigmoid\n",
    "    nn.Sigmoid(),\n",
    "    # 平均池化：池化大小为2*2，步幅为2\n",
    "    nn.AvgPool2d(kernel_size = 2,stride = 2),\n",
    "    # 卷积运算：6个输入通道，16个输出通道，5*5卷积核，无填充\n",
    "    nn.Conv2d(6,16,kernel_size=5,padding=0),\n",
    "    # 激活函数为Sigmoid\n",
    "    nn.Sigmoid(),\n",
    "    # 平均池化：池化大小为2*2，步幅为2\n",
    "    nn.AvgPool2d(kernel_size = 2,stride = 2),\n",
    "    # 展平层\n",
    "    nn.Flatten(),\n",
    "    # 全连接层：输入为16通道*高5*宽5，输出为120个神经元\n",
    "    nn.Linear(16*5*5,120),\n",
    "    # 激活函数为Sigmoid\n",
    "    nn.Sigmoid(),\n",
    "    # 全连接层：输入为120个神经元，输出为84个神经元\n",
    "    nn.Linear(120,84),\n",
    "    # 激活函数为Sigmoid\n",
    "    nn.Sigmoid(),\n",
    "    # 全连接层：输入为84个神经元，输出为10个神经元（10个分类）\n",
    "     nn.Linear(84,10 )\n",
    ")    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dadd23f",
   "metadata": {
    "origin_pos": 5
   },
   "source": [
    "我们对原始模型做了一点小改动，去掉了最后一层的高斯激活。除此之外，这个网络与最初的LeNet-5一致。\n",
    "\n",
    "下面，我们将一个大小为$28 \\times 28$的单通道（黑白）图像通过LeNet。通过在每一层打印输出的形状，我们可以[**检查模型**]，以确保其操作与我们期望的 :numref:`img_lenet_vert`一致。\n",
    "\n",
    "![LeNet 的简化版。](../img/lenet-vert.svg)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "0533e40d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T17:38:38.407712Z",
     "iopub.status.busy": "2022-12-07T17:38:38.407189Z",
     "iopub.status.idle": "2022-12-07T17:38:38.419230Z",
     "shell.execute_reply": "2022-12-07T17:38:38.418386Z"
    },
    "origin_pos": 7,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Conv2d output shape: \t torch.Size([1, 6, 28, 28])\n",
      "Sigmoid output shape: \t torch.Size([1, 6, 28, 28])\n",
      "AvgPool2d output shape: \t torch.Size([1, 6, 14, 14])\n",
      "Conv2d output shape: \t torch.Size([1, 16, 10, 10])\n",
      "Sigmoid output shape: \t torch.Size([1, 16, 10, 10])\n",
      "AvgPool2d output shape: \t torch.Size([1, 16, 5, 5])\n",
      "Flatten output shape: \t torch.Size([1, 400])\n",
      "Linear output shape: \t torch.Size([1, 120])\n",
      "Sigmoid output shape: \t torch.Size([1, 120])\n",
      "Linear output shape: \t torch.Size([1, 84])\n",
      "Sigmoid output shape: \t torch.Size([1, 84])\n",
      "Linear output shape: \t torch.Size([1, 10])\n"
     ]
    }
   ],
   "source": [
    "# 生成一张灰度图像X，检查网络各层的形状\n",
    "X = torch.rand(size=(1,1,28,28),dtype=torch.float32)\n",
    "# 逐层输出：层名，形状\n",
    "for layer in net:\n",
    "    X = layer(X)\n",
    "    print(layer.__class__.__name__,'output shape: \\t',X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d8fc7d",
   "metadata": {
    "origin_pos": 10
   },
   "source": [
    "请注意，在整个卷积块中，与上一层相比，每一层特征的高度和宽度都减小了。\n",
    "第一个卷积层使用2个像素的填充，来补偿$5 \\times 5$卷积核导致的特征减少。\n",
    "相反，第二个卷积层没有填充，因此高度和宽度都减少了4个像素。\n",
    "随着层叠的上升，通道的数量从输入时的1个，增加到第一个卷积层之后的6个，再到第二个卷积层之后的16个。\n",
    "同时，每个汇聚层的高度和宽度都减半。最后，每个全连接层减少维数，最终输出一个维数与结果分类数相匹配的输出。\n",
    "\n",
    "## 模型训练\n",
    "\n",
    "现在我们已经实现了LeNet，让我们看看[**LeNet在Fashion-MNIST数据集上的表现**]。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "65e22e22",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T17:38:38.422719Z",
     "iopub.status.busy": "2022-12-07T17:38:38.422177Z",
     "iopub.status.idle": "2022-12-07T17:38:38.507991Z",
     "shell.execute_reply": "2022-12-07T17:38:38.507133Z"
    },
    "origin_pos": 11,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "# 批大小batch_size为256\n",
    "batch_size = 256\n",
    "\n",
    "# 加载fashion_mnist，获得训练集和测试集的迭代器\n",
    "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2244eec",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "69c5767b",
   "metadata": {},
   "source": [
    "## 使用GPU训练"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "262c6e58",
   "metadata": {
    "origin_pos": 12
   },
   "source": [
    "虽然卷积神经网络的参数较少，但与深度的多层感知机相比，它们的计算成本仍然很高，因为每个参数都参与更多的乘法。\n",
    "通过使用GPU，可以用它加快训练。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3f3efb7",
   "metadata": {
    "origin_pos": 13,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "为了进行评估，我们需要[**对**] 第3章中描述的(**`evaluate_accuracy`函数进行轻微的修改**)。  \n",
    "\n",
    "由于完整的数据集位于内存中，因此在模型使用GPU计算数据集之前，我们需要将其复制到显存中。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "2d4c7017",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T17:38:38.511806Z",
     "iopub.status.busy": "2022-12-07T17:38:38.511369Z",
     "iopub.status.idle": "2022-12-07T17:38:38.518194Z",
     "shell.execute_reply": "2022-12-07T17:38:38.517458Z"
    },
    "origin_pos": 15,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def evaluate_accuracy_gpu(net, data_iter, device=None): #@save\n",
    "    \"\"\"使用GPU计算模型在数据集上的精度\"\"\"\n",
    "    # 1.设置为评估模式\n",
    "    if isinstance(net, nn.Module):\n",
    "        net.eval()  # 设置为评估模式\n",
    "        if not device:  # 没有GPU\n",
    "            device = next(iter(net.parameters())).device  # 获取CPU\n",
    "    \n",
    "    # 2.准备收集 正确预测的数量，总预测的数量\n",
    "    metric = d2l.Accumulator(2)\n",
    "    \n",
    "    # 3.计算预测值，精度等\n",
    "    with torch.no_grad():\n",
    "        for X, y in data_iter:\n",
    "            X = X.to(device)  # 将X移动到设备\n",
    "            y = y.to(device)  # 将y移动到设备\n",
    "            metric.add(d2l.accuracy(net(X), y), y.numel())  # 计算预测准确的数量，计算总个数\n",
    "    return metric[0] / metric[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4809dc2",
   "metadata": {
    "origin_pos": 17
   },
   "source": [
    "[**为了使用GPU，我们还需要一点小改动**]。  \n",
    "\n",
    "如下所示，训练函数`train_ch6`也类似于第3章中定义的`train_ch3`，但在进行正向和反向传播之前，我们需要将每一小批量数据移动到我们指定的设备（例如GPU）上。。\n",
    "- 由于我们将实现多层神经网络，因此我们将主要使用高级API。\n",
    "- 我们使用在第四章中介绍的Xavier随机初始化模型参数。\n",
    "- 与全连接层一样，我们使用交叉熵损失函数和小批量随机梯度下降。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "e5d69eab",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T17:38:38.521695Z",
     "iopub.status.busy": "2022-12-07T17:38:38.521093Z",
     "iopub.status.idle": "2022-12-07T17:38:38.531033Z",
     "shell.execute_reply": "2022-12-07T17:38:38.530266Z"
    },
    "origin_pos": 19,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def train_ch6(net, train_iter, test_iter, num_epochs, lr, device):  # 该函数在第7章继续使用\n",
    "    \"\"\"用GPU训练模型(在第六章定义)\"\"\"\n",
    "    # 1.对全连接层或卷积层，使用xavier初始化权重w\n",
    "    def init_weights(m):  \n",
    "        if type(m) == nn.Linear or type(m) == nn.Conv2d:\n",
    "            nn.init.xavier_uniform_(m.weight)\n",
    "    net.apply(init_weights)\n",
    "    \n",
    "    # 2.网络模型net移动到device设备\n",
    "    print('training on', device)\n",
    "    net.to(device)\n",
    "    \n",
    "    # 3.优化算法，损失函数\n",
    "    # 小批量随机梯度下降法\n",
    "    optimizer = torch.optim.SGD(net.parameters(),lr=lr)\n",
    "    # 交叉熵损失函数\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "    # 4.绘图用\n",
    "    animator = d2l.Animator(xlabel='epoch', xlim=[1, num_epochs],\n",
    "                            legend=['train loss', 'train acc', 'test acc'])\n",
    "    timer, num_batches = d2l.Timer(), len(train_iter)\n",
    "    \n",
    "    # 5.  每一轮\n",
    "    for epoch in range(num_epochs):  \n",
    "        # 5.1 准备收集三个值：训练损失之和，训练准确率之和，样本数\n",
    "        metric = d2l.Accumulator(3)\n",
    "        # 5.2 网络net设置为训练模式\n",
    "        net.train()\n",
    "        # 5.3 每一小批\n",
    "        for i, (X, y) in enumerate(train_iter):\n",
    "            # （1）训练\n",
    "            # 开始计时\n",
    "            timer.start()\n",
    "            # 梯度清零\n",
    "            optimizer.zero_grad()\n",
    "            # X,y移动到device\n",
    "            X = X.to(device)\n",
    "            y=y.to(device)\n",
    "            # 将X送入net，计算y_hat\n",
    "            y_hat=net(X)\n",
    "            # 根据y_hat和y，计算损失函数值l\n",
    "            l = loss(y_hat,y)\n",
    "            # 开始反向传播，计算梯度\n",
    "            l.backward()\n",
    "            # 优化参数\n",
    "            optimizer.step()\n",
    "            # （2）收集损失函数值和精度\n",
    "            with torch.no_grad():\n",
    "                metric.add(l * X.shape[0], d2l.accuracy(y_hat, y), X.shape[0]) # X.shape[0]：样本数\n",
    "            timer.stop()\n",
    "            train_l = metric[0] / metric[2]   # 训练损失均值\n",
    "            train_acc = metric[1] / metric[2] # 训练精度均值\n",
    "            if (i + 1) % (num_batches // 5) == 0 or i == num_batches - 1:  # 每轮只收集5次精度值（绘制5次）\n",
    "                animator.add(epoch + (i + 1) / num_batches, (train_l, train_acc, None))\n",
    "                \n",
    "        # 5.4 计算本轮的精度，并收集\n",
    "        test_acc = evaluate_accuracy_gpu(net, test_iter)  # 测试精度\n",
    "        animator.add(epoch + 1, (None, None, test_acc))\n",
    "        \n",
    "    # 6.输出最终的训练结果        \n",
    "    print(f'loss {train_l:.3f}, train acc {train_acc:.3f}, '\n",
    "          f'test acc {test_acc:.3f}')\n",
    "    print(f'{metric[2] * num_epochs / timer.sum():.1f} examples/sec '\n",
    "          f'on {str(device)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17b780c5",
   "metadata": {
    "origin_pos": 22
   },
   "source": [
    "现在，我们[**训练和评估LeNet-5模型**]。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "14c7fe72",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T17:38:38.534479Z",
     "iopub.status.busy": "2022-12-07T17:38:38.533859Z",
     "iopub.status.idle": "2022-12-07T17:39:22.009754Z",
     "shell.execute_reply": "2022-12-07T17:39:22.008833Z"
    },
    "origin_pos": 23,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "# 设置超参数学习率lr\n",
    "lr = 0.9\n",
    "# 设置总轮数\n",
    "num_epochs = 50\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "075b41db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"238.965625pt\" height=\"183.35625pt\" viewBox=\"0 0 238.965625 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-04-09T09:39:38.625317</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.7.0, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 238.965625 183.35625 \n",
       "L 238.965625 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 30.103125 145.8 \n",
       "L 225.403125 145.8 \n",
       "L 225.403125 7.2 \n",
       "L 30.103125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 65.974554 145.8 \n",
       "L 65.974554 7.2 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m93f77b049f\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m93f77b049f\" x=\"65.974554\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(59.612054 160.398438) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 105.831696 145.8 \n",
       "L 105.831696 7.2 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m93f77b049f\" x=\"105.831696\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(99.469196 160.398438) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 145.688839 145.8 \n",
       "L 145.688839 7.2 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m93f77b049f\" x=\"145.688839\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(139.326339 160.398438) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 185.545982 145.8 \n",
       "L 185.545982 7.2 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m93f77b049f\" x=\"185.545982\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(179.183482 160.398438) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 225.403125 145.8 \n",
       "L 225.403125 7.2 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m93f77b049f\" x=\"225.403125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 50 -->\n",
       "      <g transform=\"translate(219.040625 160.398438) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(112.525 174.076563) scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 30.103125 145.030702 \n",
       "L 225.403125 145.030702 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m1c4708c993\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1c4708c993\" x=\"30.103125\" y=\"145.030702\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.0 -->\n",
       "      <g transform=\"translate(7.2 148.829921) scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 30.103125 117.37719 \n",
       "L 225.403125 117.37719 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1c4708c993\" x=\"30.103125\" y=\"117.37719\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.5 -->\n",
       "      <g transform=\"translate(7.2 121.176409) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 30.103125 89.723678 \n",
       "L 225.403125 89.723678 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1c4708c993\" x=\"30.103125\" y=\"89.723678\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 1.0 -->\n",
       "      <g transform=\"translate(7.2 93.522897) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 30.103125 62.070165 \n",
       "L 225.403125 62.070165 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1c4708c993\" x=\"30.103125\" y=\"62.070165\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 1.5 -->\n",
       "      <g transform=\"translate(7.2 65.869384) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 30.103125 34.416653 \n",
       "L 225.403125 34.416653 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1c4708c993\" x=\"30.103125\" y=\"34.416653\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 2.0 -->\n",
       "      <g transform=\"translate(7.2 38.215872) scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_21\">\n",
       "    <path d=\"M 26.914554 13.5 \n",
       "L 27.711696 15.349235 \n",
       "L 28.508839 16.007274 \n",
       "L 29.305982 16.349787 \n",
       "L 30.103125 16.572888 \n",
       "L 30.900268 17.551332 \n",
       "L 31.697411 17.570993 \n",
       "L 32.494554 17.622407 \n",
       "L 33.291696 17.741617 \n",
       "L 34.088839 19.742826 \n",
       "L 34.885982 63.309042 \n",
       "L 35.683125 71.916842 \n",
       "L 36.480268 76.891077 \n",
       "L 37.277411 79.749145 \n",
       "L 38.074554 82.570887 \n",
       "L 38.871696 95.993454 \n",
       "L 39.668839 97.426069 \n",
       "L 40.465982 98.504427 \n",
       "L 41.263125 99.618931 \n",
       "L 42.060268 100.395345 \n",
       "L 42.857411 104.53652 \n",
       "L 43.654554 106.159922 \n",
       "L 44.451696 106.585288 \n",
       "L 45.248839 106.619748 \n",
       "L 46.045982 106.966832 \n",
       "L 46.843125 108.992924 \n",
       "L 47.640268 109.693775 \n",
       "L 48.437411 110.112226 \n",
       "L 49.234554 110.337973 \n",
       "L 50.031696 110.647543 \n",
       "L 50.828839 112.322286 \n",
       "L 51.625982 112.578121 \n",
       "L 52.423125 113.003996 \n",
       "L 53.220268 113.046136 \n",
       "L 54.017411 113.337931 \n",
       "L 54.814554 113.761171 \n",
       "L 55.611696 114.840067 \n",
       "L 56.408839 115.179379 \n",
       "L 57.205982 115.279339 \n",
       "L 58.003125 115.587128 \n",
       "L 58.800268 116.669856 \n",
       "L 59.597411 116.614687 \n",
       "L 60.394554 116.554692 \n",
       "L 61.191696 116.873259 \n",
       "L 61.988839 117.070644 \n",
       "L 62.785982 118.191922 \n",
       "L 63.583125 118.458406 \n",
       "L 64.380268 118.474045 \n",
       "L 65.177411 118.47679 \n",
       "L 65.974554 118.60836 \n",
       "L 66.771696 119.239611 \n",
       "L 67.568839 119.36943 \n",
       "L 68.365982 119.485608 \n",
       "L 69.163125 119.437417 \n",
       "L 69.960268 119.574544 \n",
       "L 70.757411 120.304308 \n",
       "L 71.554554 120.374281 \n",
       "L 72.351696 120.560546 \n",
       "L 73.148839 120.594989 \n",
       "L 73.945982 120.663835 \n",
       "L 74.743125 121.594494 \n",
       "L 75.540268 121.236008 \n",
       "L 76.337411 121.155692 \n",
       "L 77.134554 121.180474 \n",
       "L 77.931696 121.440845 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_22\">\n",
       "    <path d=\"M 26.914554 139.257296 \n",
       "L 27.711696 139.33544 \n",
       "L 28.508839 139.433502 \n",
       "L 29.305982 139.43082 \n",
       "L 30.103125 139.469581 \n",
       "L 30.900268 139.496323 \n",
       "L 31.697411 139.441163 \n",
       "L 32.494554 139.386003 \n",
       "L 33.291696 139.125142 \n",
       "L 34.088839 137.839867 \n",
       "L 34.885982 122.153121 \n",
       "L 35.683125 119.077955 \n",
       "L 36.480268 117.274531 \n",
       "L 37.277411 116.360179 \n",
       "L 38.074554 115.062591 \n",
       "L 38.871696 109.277874 \n",
       "L 39.668839 108.535513 \n",
       "L 40.465982 108.090403 \n",
       "L 41.263125 107.656401 \n",
       "L 42.060268 107.381367 \n",
       "L 42.857411 105.476435 \n",
       "L 43.654554 104.881167 \n",
       "L 44.451696 104.768549 \n",
       "L 45.248839 104.780041 \n",
       "L 46.045982 104.654731 \n",
       "L 46.843125 103.99631 \n",
       "L 47.640268 103.619384 \n",
       "L 48.437411 103.378825 \n",
       "L 49.234554 103.240159 \n",
       "L 50.031696 103.103369 \n",
       "L 50.828839 102.585135 \n",
       "L 51.625982 102.313932 \n",
       "L 52.423125 102.108615 \n",
       "L 53.220268 102.149602 \n",
       "L 54.017411 102.033178 \n",
       "L 54.814554 101.799106 \n",
       "L 55.611696 101.355528 \n",
       "L 56.408839 101.166299 \n",
       "L 57.205982 101.141784 \n",
       "L 58.003125 100.975892 \n",
       "L 58.800268 100.466074 \n",
       "L 59.597411 100.574096 \n",
       "L 60.394554 100.596313 \n",
       "L 61.191696 100.429301 \n",
       "L 61.988839 100.383185 \n",
       "L 62.785982 99.850122 \n",
       "L 63.583125 99.840929 \n",
       "L 64.380268 99.735205 \n",
       "L 65.177411 99.748995 \n",
       "L 65.974554 99.684473 \n",
       "L 66.771696 99.353683 \n",
       "L 67.568839 99.270943 \n",
       "L 68.365982 99.275539 \n",
       "L 69.163125 99.305418 \n",
       "L 69.960268 99.245704 \n",
       "L 70.757411 99.077883 \n",
       "L 71.554554 98.894016 \n",
       "L 72.351696 98.837324 \n",
       "L 73.148839 98.827365 \n",
       "L 73.945982 98.799561 \n",
       "L 74.743125 98.301047 \n",
       "L 75.540268 98.450439 \n",
       "L 76.337411 98.486446 \n",
       "L 77.134554 98.532029 \n",
       "L 77.931696 98.448361 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_23\">\n",
       "    <path d=\"M 30.103125 139.5 \n",
       "L 34.088839 122.797279 \n",
       "L 38.074554 113.090896 \n",
       "L 42.060268 107.361088 \n",
       "L 46.045982 103.473004 \n",
       "L 50.031696 103.28496 \n",
       "L 54.017411 102.217535 \n",
       "L 58.003125 102.322618 \n",
       "L 61.988839 100.016315 \n",
       "L 65.974554 101.122456 \n",
       "L 69.960268 99.302854 \n",
       "L 73.945982 100.547263 \n",
       "L 77.931696 98.722131 \n",
       "\" clip-path=\"url(#p9c4b1ad48d)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 30.103125 145.8 \n",
       "L 30.103125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 225.403125 145.8 \n",
       "L 225.403125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 30.103125 145.8 \n",
       "L 225.403125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 30.103125 7.2 \n",
       "L 225.403125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 140.634375 59.234375 \n",
       "L 218.403125 59.234375 \n",
       "Q 220.403125 59.234375 220.403125 57.234375 \n",
       "L 220.403125 14.2 \n",
       "Q 220.403125 12.2 218.403125 12.2 \n",
       "L 140.634375 12.2 \n",
       "Q 138.634375 12.2 138.634375 14.2 \n",
       "L 138.634375 57.234375 \n",
       "Q 138.634375 59.234375 140.634375 59.234375 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_24\">\n",
       "     <path d=\"M 142.634375 20.298438 \n",
       "L 152.634375 20.298438 \n",
       "L 162.634375 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- train loss -->\n",
       "     <g transform=\"translate(170.634375 23.798438) scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"292.333984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"353.515625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"405.615234\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_25\">\n",
       "     <path d=\"M 142.634375 34.976563 \n",
       "L 152.634375 34.976563 \n",
       "L 162.634375 34.976563 \n",
       "\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- train acc -->\n",
       "     <g transform=\"translate(170.634375 38.476563) scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"325.830078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"380.810547\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_26\">\n",
       "     <path d=\"M 142.634375 49.654688 \n",
       "L 152.634375 49.654688 \n",
       "L 162.634375 49.654688 \n",
       "\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- test acc -->\n",
       "     <g transform=\"translate(170.634375 53.154688) scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"192.041016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"223.828125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"285.107422\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"340.087891\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p9c4b1ad48d\">\n",
       "   <rect x=\"30.103125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ---> 开始训练\n",
    "train_ch6(net, train_iter, test_iter, num_epochs, lr, d2l.try_gpu())  # 本地cpu可以运行，约5分钟结束"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29d7aba2",
   "metadata": {
    "origin_pos": 24
   },
   "source": [
    "## 小结\n",
    "\n",
    "* 卷积神经网络（CNN）是一类使用卷积层的网络。\n",
    "* 在卷积神经网络中，我们组合使用卷积层、非线性激活函数和汇聚层。\n",
    "* 为了构造高性能的卷积神经网络，我们通常对卷积层进行排列，逐渐降低其表示的空间分辨率，同时增加通道数。\n",
    "* 在传统的卷积神经网络中，卷积块编码得到的表征在输出之前需由一个或多个全连接层进行处理。\n",
    "* LeNet是最早发布的卷积神经网络之一。\n",
    "\n",
    "## 练习\n",
    "\n",
    "1. 将平均汇聚层替换为最大汇聚层，会发生什么？\n",
    "1. 尝试构建一个基于LeNet的更复杂的网络，以提高其准确性。\n",
    "    1. 调整卷积窗口大小。\n",
    "    1. 调整输出通道的数量。\n",
    "    1. 调整激活函数（如ReLU）。\n",
    "    1. 调整卷积层的数量。\n",
    "    1. 调整全连接层的数量。\n",
    "    1. 调整学习率和其他训练细节（例如，初始化和轮数）。\n",
    "1. 在MNIST数据集上尝试以上改进的网络。\n",
    "1. 显示不同输入（例如毛衣和外套）时，LeNet第一层和第二层的激活值。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "797998b9",
   "metadata": {
    "origin_pos": 26,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "[Discussions](https://discuss.d2l.ai/t/1860)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
